Spectrally-truncated kernel ridge regression and its free lunch
نویسندگان
چکیده
Kernel ridge regression (KRR) is a well-known and popular nonparametric approach with many desirable properties, including minimax rate-optimality in estimating functions that belong to common reproducing kernel Hilbert spaces (RKHS). The approach, however, computationally intensive for large data sets, due the need operate on dense n×n matrix, where n sample size. Recently, various approximation schemes solving KRR have been considered, some analyzed. Some approaches such as Nyström sketching shown preserve rate optimality of KRR. In this paper, we consider simplest approximation, namely, spectrally truncating matrix its largest r<n eigenvalues. We derive an exact expression maximum risk truncated KRR, over unit ball RKHS. This result can be used study trade-off between level spectral truncation regularization parameter. show that, long RKHS infinite-dimensional, there threshold r, above which, spectrally-truncated surprisingly outperforms full terms risk, minimum taken strengthens existing results schemes, by showing not only one does lose rates, fact improve performance, all finite samples (above threshold). Moreover, implicit achieved substitute norm regularization. Both are needed achieve best performance.
منابع مشابه
Reduced rank ridge regression and its kernel extensions
In multivariate linear regression, it is often assumed that the response matrix is intrinsically of lower rank. This could be because of the correlation structure among the prediction variables or the coefficient matrix being lower rank. To accommodate both, we propose a reduced rank ridge regression for multivariate linear regression. Specifically, we combine the ridge penalty with the reduced...
متن کاملKernel Ridge Regression via Partitioning
In this paper, we investigate a divide and conquer approach to Kernel Ridge Regression (KRR). Given n samples, the division step involves separating the points based on some underlying disjoint partition of the input space (possibly via clustering), and then computing a KRR estimate for each partition. The conquering step is simple: for each partition, we only consider its own local estimate fo...
متن کاملDivide and Conquer Kernel Ridge Regression
We study a decomposition-based scalable approach to performing kernel ridge regression. The method is simple to describe: it randomly partitions a dataset of size N into m subsets of equal size, computes an independent kernel ridge regression estimator for each subset, then averages the local solutions into a global predictor. This partitioning leads to a substantial reduction in computation ti...
متن کاملModelling Issues in Kernel Ridge Regression
Kernel ridge regression is gaining popularity as a data-rich nonlinear forecasting tool, which is applicable in many different contexts. This paper investigates the influence of the choice of kernel and the setting of tuning parameters on forecast accuracy. We review several popular kernels, including polynomial kernels, the Gaussian kernel, and the Sinc kernel. We interpret the latter two kern...
متن کاملAn Identity for Kernel Ridge Regression
This paper derives an identity connecting the square loss of ridge regression in on-line mode with the loss of the retrospectively best regressor. Some corollaries about the properties of the cumulative loss of on-line ridge regression are also obtained.
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: Electronic Journal of Statistics
سال: 2021
ISSN: ['1935-7524']
DOI: https://doi.org/10.1214/21-ejs1873